A Two-stage Committee Machine of Neural Networks

In solving pattern recognition problems, many ensemble methods have been proposed to replace a single classifier by a classifier committee. These methods can be divided roughly into two categories: serial and parallel approaches. In the serial approach, component classifiers are created by focusing on different parts of the training set in different learning phases. In contrast, without paying special attention to any part of the dataset, the parallel approach generates classifiers independently. By integrating these two techniques and by using a neural network approach for the base classifier, this work proposes a design method for a two-stage committee machine. In the first stage of the approach the entire dataset is used to train an averaging ensemble. Based on the classification results of this first stage, hard-to-classify samples are selected and sent to the second stage. To improve the classification accuracy for these samples, a computationally more intensive bagging ensemble is employed in the second stage. These two neural network ensembles work in series whereas the component neural networks in each of the ensembles are trained in parallel. Experimental results demonstrate the accuracy and robustness of the proposed approach.